High dynamic range imaging (HDRI) is a term applied in image processing, computer graphics and photography, and generally relates to systems or techniques for providing a greater dynamic range of exposures. HDRI is most commonly employed in situations where the range between light and dark areas is great, and subsequently a normal exposure, or even a digitally enhanced exposure, are not adequate to resolve all of the image area.
HDRI manipulates images and exposures to accurately represent the wide range of intensity levels found in real scenes, from direct sunlight to shadows. With HDRI, the user employs multiple exposures and bracketing with photo merging, to get greater detail throughout the tonal range.
More particularly, HDRI processing involves merging several exposures of a given scene into a, typically, 32-bit HDRI source file, which is then “tone mapped” to produce an image in which adjustments of qualities of light and contrast are applied locally to the HDRI source image.
HDRI images are best captured originally in a digital format with a much higher bit depth than the current generation of digital imaging devices. Current devices are built around an 8-bit per channel architecture. That means that both the cameras and output displays have a maximum tonal range of 8-bits per RGB color channel.
HDRI formats are typically 32-bits per channel. A few next generation cameras and displays are capable of handling this kind of imagery natively. It will probably be quite a few years until HDRI displays become common but HDRI cameras and acquisition techniques are already emerging.
HDRI images are typically tone-mapped back to 8-bits per channel, essentially compressing the extended information into the smaller dynamic range. This is typically done automatically with a variety of existing software algorithms, or manually with artistic input through programs like Adobe Photoshop.
So in a typical workflow for HDRI the artist first captures the HDRI image, and then the image is tone-mapped back to desired output device such as ink on paper, an 8-bit RGB monitor, or even a 32-bit HDRI monitor (requiring no tone mapping).
The real challenge with HDRI is not the file formats or computer algorithms to tone map them to 8-bit displays. Those challenges have already been largely met. For example, open EXR is an example of a robust open source HDRI format developed by Industrial Light and Magic. The hardest part of capturing HDR images is the physical devices used to capture the imagery. So far only two ways of capturing HDR images are available.
The first is to use exotic high end cameras with special imaging chips (CMOS or CCD) like the Spheron HDR. Both CCD (charge-coupled device) and CMOS (complimentary metal-oxide semiconductor) image sensors convert light into electrons, though CMOS sensors are much less expensive to manufacture than CCD sensors. These types of cameras are typically used by professionals in controlled environments for the primary purpose of creating spherical photos to illuminate computer generated images (another important use of HDRI). They are not point and shoot cameras and are not capable of motion photography.
The second is shooting multiple varying exposures in rapid succession (known as exposure bracketing) then combining those images taking the highlights from the underexposed images, mid tones from the normally exposed images, and shadows from the over exposed images to create a composite HDR image that retains massive detail in the highlights and shadows where normal cameras would lose detail.
Both of these techniques have substantial disadvantages. The second technique can be done with conventional hardware, but it is time consuming and takes substantial expertise to pull off. In addition, because the images are not temporally aligned, meaning they were taken one after another at different moments in time, there can be changes in the scene that produce artifacts when the HDRI software attempts to eliminate or synthesize the objects in motion across the frame. An example would be a car moving through the frame.
Even a slight movement of the camera between exposures will be noticeable in the resulting combined image. Moving objects will be “ghosted” in the HDRI image. As such this technique is totally useless for motion photography and can only be used with substantial success in still photography applications.
For this reason, exposure bracketed HDRI is typically restricted to still subjects, and any animals, cars, pedestrians, moving leaves or litter, clouds, etc., in fact anything that is shifting within the frame will preclude HDRI, or at the very least lead to unhappy results.
Further, producing HDRI from multiple images can be a time consuming and frustrating task. HDRI requires multiple, huge files, multiple steps, and typically specialized and complicated software.
The first technique is very expensive and requires exotic hardware or sophisticated electronic and software systems. While imaging chips are moving ever forward in sensitivity and dynamic range, they still do not produce the dramatic results that the first technique of changing exposures does. In addition, these special cameras are not capable of shooting higher frame rates required to shoot motion pictures. These products are used for narrow specialized purposes.
Proposed solutions to the problems associated with the second technique are reflected in various published patents at the United States Patent and Trademark Office. For example, United States Patent Application No. 20060221209, to McGuire, et al., published Oct. 5, 2006, teaches an apparatus and method for acquiring and combining images of a scene with multiple optical characteristics at multiple resolutions. Disclosed therein is a camera system that acquires multiple optical characteristics at multiple resolutions of a scene. The camera system includes multiple optical elements arranged as a tree having a multiple of nodes connected by edges. The system employs filters at the end of the chain, and lenses are placed in front of each of the sensors, creating additional sources of optical distortion.
United States Patent Application No. 20070126918, to Lee, published Jun. 7, 2007, discloses cameras that can provide improved images by combining several shots of a scene taken with different exposure and focus levels is provided. In addition, cameras are provided, which have pixel-wise exposure control means so that high quality images are obtained for a scene with a high level of contrast. The system is complicated, and employs light reducing filters to create exposures of varying intensity. Much of the light is lost, reducing clarity and introducing sources of distortion and noise to the images.
United States Patent Application No. 20080149812, to Ward, et al., published Jun. 26, 2008, discloses an electronic camera comprising two or more image sensor arrays. At least one of the image sensor arrays has a high dynamic range. The camera also comprises a shutter for selectively allowing light to reach the two or more image sensor arrays, readout circuitry for selectively reading out pixel data from the image sensor arrays, and, a controller configured to control the shutter and the readout circuitry. The controller comprises a processor and a memory having computer-readable code embodied therein which, when executed by the processor, causes the controller to open the shutter for an image capture period to allow the two or more image sensor arrays to capture pixel data, and, read out pixel data from the two or more image sensor arrays. This is essentially a total digital solution to the problem of controlling exposure levels for different images for high dynamic range processing.
Finally, United States Patent Application No. 20070177004, to Kolehmainen, et al., published Aug. 2, 2007, is directed to an image creating method and imaging device comprising at least two image capturing apparatus, each apparatus being arranged to produce an image. The apparatus is configured to utilize at least a portion of the images produced with different image capturing apparatus with each other to produce an image with an enhanced image quality. Multiple lenses are required to implement this method, which is expensive and creates parallax and optic imagery distortions with each lens addition.
None of the prior approaches have been able to provide a simple means for capturing multiple images that overcome the difficulties of temporal misalignment, and that are simple and quickly resolved into a high definition range image.
What is needed is an inexpensive solution that can be easily integrated into products with conventional form factors. This solution would ideally be easy to use, compact, and able to shoot at high frame rates with no introduction of temporal alignment problems and associated artifacts.